Learning from experimental and observational data

Guido Biele

Background

Bias

  • Most scientific enterprises attempt to estimate some quantity or effect
  • Bias is present if our estimate of that effect systematically deviates from the true value
  • When thinking about bias, it is important to clearly define the estimand: What is it that we are trying to estimate?
    • \(bias = estimand - estimate\)

Estimate the right thing

A well described estimand is characterized by

  • a clear effect measure
    • i.e., not just “Stress”, but “Stress, operationalized as …”
  • a clear target population
    • for which set of humans do we want to measure the effect?

Internal and external validity

  • If a study has internal validity, we correctly estimate the effect of a manipulation in the study sample
  • If a study has external validity, the estimated effect in our study sample is the same as in the target population


  • Studies with either internal or external validity can still be biased
  • Internal and external validity alone are necessary but not sufficient conditions for obtaining unbiased estimates.

Bias in observational studies

Nobody needs convincing that confounding is a problem

Observed confounders are not a problem, we can simply adjust.

Unobserved confounders are a problem! There is little we can do!

Treatment randomisation prevent confounding.

Selection bias

If the treatment determines who stays in the study and there is another variable L that causes who stays and the outcome, there will be selection bias

Selection bias in obs. & exp. studies

Estimating the effect of distance to work out place on fitness will be biased if participation in the follow up (FU) depends on distance, and unobserved education influences both FU & fitness.

An RCT where medication influences side effects (Si), which influence drop out (DO), and unobserved comorbid disorders influence Si & symptoms will produce a biased effect estimate.

Experiments with drop out also require the assumption of no unobserved confounding!

Bias in experimental studies I

If there are some characteristics C that influence the effect of the treatment T on the outcome O (T-C-interaction), and the distribution of C is different in the study sample and target population, a randomised experiment will produce biased effect estimates.

“Bias” in experimental studies II

When the outcomes of interest \(O_1\) and \(O_2\) depend exclusively on a common mediator \(M\) then results about the effect on \(O_1\) will be informative about the effect on \(O_2\)

When only one of the outcomes depends on and additional variable \(X\), results from experiment with outcome \(O_1\) are a biased estimate for the effect on \(O_2\).

Translational research needs experiments and observation

  • Results from observational studies can be biased
  • But the same is true for experiments
  • Experiments can be improved through random sampling of participants and investigation of multiple outcomes
  • Even then, many experiments cannot be performed for ethical reasons

Observational research is hard

Does parental income influence children’s mental health?

Kinge et al., Parental income and mental disorders in children and adolescents: prospective register-based study. IJE (2021).
“…associations between lower parental income and children’s mental disorders were partly, but not fully, attributed to other socio-demographic factors…”

Results of Kinge et al. 2021 IJE

Does parental income influence children’s mental health?

Sariaslan et al., No causal associations between childhood family income and subsequent psychiatric disorders, substance misuse and violent crime arrests: a nationwide Finnish study of >650 000 individuals and their siblings. IJE (2021)

Results of Sariaslan et al. 2021 IJE

Study designs

We analyse one data set to see in how far the results of Sariaslan and Kinge depend on modelling assumptions.
When the outcome is binary, sibling designs and panel designs with FE can only use a subset of data: Siblings and individuals with discordant outcomes, respectively.

Samples sizes and prevalence

Choosing a particular study design introduces substantial differences between the study sample and the source and target population.

Results: Odds & hazard ratios

Association is stronger when parents do not already have a diagnosis.

Results: Odds & hazard ratios

Original results of Kinge at al (2021) and Sariaslan et al (2021) can be replicated in this data set.

Results: Odds & hazard ratios

Only a logistic (or cox) regression with adjustment finds an association.

Summary of results

  • We can replicate the results of Kinge et al. (there is an association) and Sariaslan et al. (there is no association) using the same basic data set
  • Conditional logit regression also shows no effect

Sibling design and fixed effects panel model make fewer assumptions / have a higher internal validity. Should we rather trust these results?

Income variability

Income variation in designs with high internal validity is only a fraction (~20%) of the income variability between families.

Selection bias

If parental income is a cause of # children & we are looking for a negative effect of income, sibling designs show a biased (attenuated) association. Sibling design studies have higher internal validity lower but external validity.

Summary

  • Different results of Kinge et al. (2021) and Sariaslan et al. (2021) are likely due to the choice of different analysis approaches
  • There is no categorical difference between “non-causal” and “causal” designs.
  • Causal inference always needs assumptions, which should be scrutinized.

Conclusion

  • Causal inference is hard, RCTs don’t guarantee unbiasedness
    • randomized treatment assignment is not enough
  • Especially applied research needs observational studies
  • Causal inference with observational data is harder
    • But: Difference in difference, Regression discontinuity, Instrumental variables, Sibling designs …
    • “Causal designs” are no panacea!
  • Sensitivity analysis: What do the results look like if we assume some unobserved confounding